Method and system for processing checksum of a data stream to optimize deduplication

ABSTRACT

Techniques for deduplicating a data stream with checksum data embedded therein are described. According to one embodiment, a first data stream is received from a client having a plurality of data regions and a plurality of checksums for verifying integrity of the data regions embedded therein, where the first data stream represents a file or a directory of one or more files of a file system associated with the client. In response the first data stream with the checksums removed is deduplicated into a plurality of deduplicated chunks.

RELATED APPLICATIONS

This application is related to co-pending U.S. patent application Ser.No. 13/718,827, entitled “Method and System for Handling ObjectBoundaries of a Data Stream to Optimize Deduplication,” filed Dec. 18,2012.

FIELD OF THE INVENTION

Embodiments of the present invention relate generally to data storagesystems. More particularly, embodiments of the invention relate toprocessing checksum of a data stream to optimize data for betterdeduplication.

BACKGROUND

Data storage utilization is continually increasing, causing theproliferation of storage systems in data centers. In order to reducestorage space of a storage system, deduplication techniques areutilized, where data objects or files are segmented in chunks and onlythe deduplicated chunks are stored in the storage system.

At the time of data recovery by clients, there is a need to validatedata integrity as data are read back from a storage system (e.g., backupstorage system). Some data would include integrity verification datasuch as checksum data inside the data stream for data integrityverification during the restoration. However, such integrityverification data may cause seriously degraded deduplication of theactual data stream at the storage system. Such data integrity checkshould survive backup data migration among different tiers of storage,while its operation efficiency should be guaranteed with random dataaccess and deduplication effectiveness at the storage systems preserved.

In addition, a data stream such as a backup data stream typicallyconsists of a sequence of data objects or files. Typically, storagesystems are unaware of individual file boundaries that can be used asheuristics for segmentation, thus segmentation boundaries tend torandomize across file boundaries. As a result, such a configurationwould affect performance and/or deduplication effectiveness, because afile is often represented and accessed as a whole unit during backup andsynthetic operations.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example and notlimitation in the figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1 is a block diagram illustrating a storage system according to oneembodiment of the invention.

FIG. 2 a block diagram illustrating a marker data structure according toone embodiment of the invention.

FIG. 3 is a flow diagram illustrating a method for deduplicationaccording to one embodiment of the invention.

FIG. 4 is a block diagram illustrating an example of a data streamhaving markers embedded therein according to one embodiment of theinvention.

FIG. 5 is a block diagram illustrating a process of handling checksummarkers according to one embodiment of the invention.

FIGS. 6A and 6B are flow diagrams illustrating a method for processingchecksum markers according to certain embodiments of the invention.

FIG. 7 is a block diagram illustrating a process for handling boundarymarkers according to one embodiment of the invention.

FIGS. 8A and 8B are flow diagrams illustrating a method for processingboundary markers according to certain embodiments of the invention.

FIG. 9 is a block diagram illustrating a deduplicated storage systemaccording to one embodiment of the invention.

DETAILED DESCRIPTION

Various embodiments and aspects of the inventions will be described withreference to details discussed below, and the accompanying drawings willillustrate the various embodiments. The following description anddrawings are illustrative of the invention and are not to be construedas limiting the invention. Numerous specific details are described toprovide a thorough understanding of various embodiments of the presentinvention. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments of the present inventions.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin conjunction with the embodiment can be included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the specification do not necessarilyall refer to the same embodiment.

According to some embodiments, storage software (e.g., backup software)is equipped with knowledge about the structure and nature of the databeing stored. Such application specific knowledge can provide importantheuristics for backend storage processing such as deduplicationprocessing. In one embodiment, special markers are embedded into a datastream (e.g., backup data stream) to transfer such data specificknowledge to backend storage systems to be recognized by the storagesystems. As a result, certain optimization processes can be performedduring storage processes such as deduplication processes, whilemaintaining desired functionality for the clients.

In one embodiment, checksum markers are embedded in a data stream thatincludes multiple data regions. Each checksum marker is associated witheach of the data regions and each checksum marker includes or identifiesa checksum within the data stream for verifying the integrity of thecorresponding data region. The checksum markers may be inserted into thedata stream by a client (e.g., backup software client) interleaving withthe corresponding data regions. The data stream is then transmitted to aremote storage system (e.g., backend storage system such as a backupstorage system) for storage. The checksum markers may be utilized forverifying the integrity of the data regions subsequently when the datastream is restored from the remote storage system. When the storagesystem receives the data stream having checksum markers therein, thestorage system parses the data stream to recognize or identify thechecksum markers and to remove the checksum markers from the datastream. The data stream having the checksum markers removed is thendeduplicated to generate deduplicated chunks (also referred to asdeduplicated segments). The deduplicated chunks are then stored in thestorage system. In addition, the checksum markers are separately stored,which may be used to reconstruct the data stream subsequently during therestoration. As a result, the checksums of the data regions within thedata stream would not significantly affect the deduplication efficiency.

According to another embodiment, another type of markers is utilized toidentify boundaries of data objects within the data stream. When thestorage system is to perform a deduplication process, the storage systemcan segment the data stream into segments using a chunking algorithm.When segmenting the data stream, the storage system can recognize theboundary markers and take the data object boundaries into theconsideration in deciding where to anchor the data stream. As a result,a better deduplication and data access efficiency can be achieved. Boththe checksum markers and boundary markers can be utilized individuallyor in combination within a data stream.

FIG. 1 is a block diagram illustrating a storage system according to oneembodiment of the invention. Referring to FIG. 1, system 100 includes,but is not limited to, one or more client systems 101-102communicatively coupled to storage system 104 over network 103. Clients101-102 may be any type of clients such as a server, a personal computer(e.g., desktops, laptops, and tablets), a “thin” client, a personaldigital assistant (PDA), a Web enabled appliance, a gaming device, amedia player, or a mobile phone (e.g., Smartphone), etc. Network 103 maybe any type of networks such as a local area network (LAN), a wide areanetwork (WAN) such as the Internet, or a combination thereof.

Storage system 104 may represent any type of server or cluster ofservers. For example, storage system 104 may be a storage server usedfor any of various different purposes, such as to provide users withaccess to shared data and/or to back up data such as mission criticaldata. In one embodiment, storage system 104 includes, but is not limitedto, backup engine 106, deduplication storage engine 107, and one or morestorage units 108-109 communicatively coupled to each other. Storageunits 108-109 may be implemented locally (e.g., single node operatingenvironment) or remotely (e.g., multi-node operating environment) viainterconnect 120, which may be a bus and/or a network. Backup engine 106is configured to back up data of clients 101-102 and to store the backupfiles in storage units 108-109.

In response to a data file, for example, received from backup engine106, to be stored in storage units 108-109, according to one embodiment,deduplication storage engine 107 is configured to segment the data fileinto multiple chunks (also referred to as segments) according to avariety of segmentation policies or rules. For example, a file may bebroken into chunks by identifying chunk boundaries using a content-basedtechnique. In one particular embodiment, a function is calculated atvarious locations of a file, when the function is equal to a value orwhen the value is a minimum, a maximum, or other value relative to otherfunction values calculated for the file. Alternatively, a file may besegmented into chunks by identifying chunk boundaries using anon-content-based technique (e.g., based on size of the chunk). In oneembodiment, a chunk is restricted to a minimum and/or maximum length, toa minimum or maximum number of chunks per file, or any other appropriatelimitation. Deduplication storage engine 107 may choose not to store achunk in a storage unit if the chunk has been previously stored instorage units 108-109. In the event that deduplication storage engine107 chooses not to store the chunk in storage units 108-109, it storesmetadata associated with the chunk to enable the reconstruction of thefile using the previously stored chunk. As a result, chunks of datafiles are stored in a deduplicated manner, either within each of storageunits 108-109 or across at least some of storage units 108-109. Themetadata, such as metadata 110-111, may be stored in at least some ofstorage units 108-109, such that files can be accessed independent ofanother storage unit. Metadata of each storage unit includes enoughinformation to provide access to the files it contains.

According to one embodiment, backup engine 106 includes a markerprocessing unit 115 to process markers of a backup stream received fromclients 101-102. In one embodiment, marker processing unit 115 is torecognize the markers marking certain data, such as checksum data ordata object boundaries within the backup stream. Once the markers havebeen recognized, the associated data is identified based on the markers.For example, the size or a boundary of checksum data or a data regionmay be determined based on the markers. The markers and/or associateddata (e.g., the actual checksum data) may be extracted or removed fromthe data stream. The remaining data stream (without the markers and/orthe associated data) may be deduplicated by deduplication storage engine107 into deduplicated data chunks in view of the information provided bythe markers. The deduplicated data chunks are then stored in any ofstorage units 108-109 as part of data objects 112-113. The markersand/or the associated data may also be separately stored in storageunits 108-109. In one embodiment, the markers and/or associated data maybe inserted into the backup stream at the client prior to transmittingthe backup stream to storage system 104.

For example, a backup stream may be generated by backup client software105 of client 101 in response to a request for backing up data stored inclient 101. The backup stream may represent a file or a directory of oneor more files or subdirectories, dependent upon the specific backuppolicy or schedule. In addition, when generating the data stream, backupclient software 105 is to insert markers and/or associated checksum datainto the data stream prior to transmitting the data stream to storagesystem 104, where the markers will be recognized by marker processingunit 115 to perform additional processes based on the informationprovided by the markers.

In one embodiment, a data stream includes multiple data regions (e.g.,fixed-size regions). When a data stream is being formed, client software105 is to generate a checksum for each data region and insert a checksummarker (also referred to as a marker header) in front of each checksumdata block to identify the corresponding checksum data block. Accordingto one embodiment, a marker header includes a predetermined pattern(signature) and a length identifying a size of the marker data blockfollowing the marker header, as shown in FIG. 2. According to anotherembodiment, client software 105 may also identify the boundaries of thedata regions and insert boundary markers into the data stream during theconstruction of the data stream. Note that although not shown, thearchitecture of clients 101-102 may be identical or similar. Client 102may also include backup client software and its corresponding markingunit running therein.

When the backup stream is received by backup engine 106 from backupclient software 105, marker processing unit 115 of backup engine 106 isto scan the backup stream to recognize the markers inserted by clientsoftware 105 to identify the corresponding marker. During theconstruction of a backup stream, the backup stream may be partitionedinto multiple data regions by client software 105. A checksum isgenerated for each region and a checksum marker is inserted into thebackup stream to be associated with the corresponding data region forthe purpose of verifying integrity of the corresponding data region. Inaddition, client software 105 further inserts a checksum marker toidentify the checksum for each data region.

The checksum markers may be utilized for verifying the integrity of thedata regions subsequently when the data stream is restored from theremote storage system. When the storage system 104 receives the datastream having checksum markers therein, marker processing unit 115parses the data stream to recognize or identify the checksum markers andto remove the checksum markers and/or the associated checksum data fromthe data stream. The data stream having the checksum markers removed isthen deduplicated by deduplication storage engine 107 to generatededuplicated chunks (also referred to as deduplicated segments). Thededuplicated chunks are then stored in storage units 108-109 of storagesystem 104. In addition, the checksum markers are separately stored,which may be used to reconstruct the data stream subsequently during therestoration. As a result, the checksums of the data regions within thedata stream would not significantly affect the deduplication efficiency.

According to another embodiment, another type of marker is utilized toidentify boundaries of data objects within the data stream. Whendeduplication storage engine 107 is to perform a deduplication process,deduplication storage engine 107 can segment the data stream intosegments using a chunking algorithm. When segmenting the data stream,deduplication storage engine 107 can recognize the boundary markers andtake the data object boundaries into the consideration in deciding whereto anchor the data stream for the purpose of segmenting the data streaminto deduplicated chunks. As a result, a better deduplication and dataaccess efficiency can be achieved. Both the checksum markers andboundary markers can be utilized individually or in combination within adata stream.

Note that marker processing unit 115 may be implemented as a standaloneprocessing unit communicatively coupled to backup engine 106 and/ordeduplication storage engine 107. In this example, storage system 104operates as a backup server. In another configuration, storage system104 may operate as a non-backup system, such as, for example, a regularfile server. When storage system 104 operates as a regular storagesystem, marker processing unit 115 can be coupled to an interface, suchas a file system interface, a Web interface, an application programminginterface (API), etc., to receive a file or files from clients 101-102to be stored in storage units 108-109. Other configurations may alsoexist.

FIG. 2 is a block diagram illustrating an example of a marker datastructure according to one embodiment. According to one embodiment,marker 200 includes a pattern field 201, a type of marker field 202, anda length field 203. Pattern field 201 is used by processing logic torecognize that this is a marker the processing logic is responsible forprocessing. Type field 202 is used to identify the type of the marker,such as, for example, a checksum marker or a boundary marker, and it canbe used to identify other types of markers. Length field 203 is used tospecify the size of the data block 210 (e.g., checksum data) thatfollows marker 200. For example, type 202 of a checksum marker stores avalue indicating that the corresponding marker is a checksum marker anddata block 210 contains the actual checksum data. Similarly, type 202 ofa boundary marker stores a different value indicating that thecorresponding marker is a boundary marker. A boundary marker may have azero value in length field 203 and it does not have marker data block210 as a result. Note that pattern field 201 and type field 202 may be asingle field to identify the marker. In one embodiment, type field 202stores a large integer to represent a particular marker. In oneparticular embodiment, the large integer may be a square of a largeprime number. Other integer numbers may also be utilized.

FIG. 3 is a flow diagram illustrating a method for improvingdeduplication using markers according to one embodiment of theinvention. Method 300 may be performed by marker processing unit 115 ofFIG. 1, which may be processing logic implemented in software, hardware,or a combination thereof. Referring to FIG. 3, at block 301, processinglogic receives a data stream such as a backup stream from a remoteclient. The data stream may represent at least a portion of files ordirectories of files of the client. At block 302, processing logic scansthe data stream to recognize one or more markers (e.g., checksum markersor boundary markers) embedded therein, where the markers may be insertedinto the data stream by the client. As described above, a marker mayinclude a marker header having a marker type field identifying a type ofthe marker as shown in FIG. 2. At block 303, processing logic identifiestype(s) of the markers, for example, based on a type field of themarkers. At block 304, certain actions associated with the identifiedtype(s) of the markers are performed. For example, processing logic mayremove the checksum markers from the data stream prior to thededuplication. However, dependent upon the specific circumstances, forsome types of markers such as boundary markers that would not have asignificant impact on the deduplication efficiency, there is no need toremove them from the data stream. Thereafter, the data stream isdeduplicated into deduplicated chunks in view of the informationprovided by the markers. The deduplicated chunks and the removed markersmay be separately stored in the storage.

FIG. 4 is a block diagram illustrating an example of a data streamhaving markers embedded therein according to one embodiment of theinvention. For example, data stream 200 may represent a backup streamgenerated by backup client software 105 and processed by backup engine106 of FIG. 1. Referring to FIG. 4, data stream 400 includes dataregions 401-403, each data region being associated with one of checksumblocks 404-406 for verifying integrity of the corresponding data region.Note that locations and format of checksum blocks 404-406 may varydependent upon the specific file systems or operating systems they areassociated with. According to one embodiment, a checksum block, in thisexample checksum block 405, includes a checksum marker in a form ofchecksum marker or header 407 and checksum data 408. Checksum marker 407may be inserted by a client such as client software 105. For example,when generating data stream 400, client software 105 calculates andinserts checksum data 408 into data stream 400. In addition, for eachchecksum inserted, client software 105 also inserts checksum marker 407,for example, in front of checksum data 408.

According to one embodiment, a checksum marker, in this example checksummarker 407, includes a checksum pattern 409 and a length 410. Checksumpattern 409 includes a predetermined pattern that can be used toidentify checksum data 408. Length 410 specifies the size of checksumdata 408. As a result, when the marker processing unit scans data stream400, it can recognize checksum marker 407 based on its checksum pattern409 and determine the size of checksum data 408 based on length 410.Checksum marker 407 can be implemented in a variety of data structures.

FIG. 5 is a block diagram illustrating a process of handling checksummarkers according to one embodiment of the invention. Referring to FIG.5, as first data stream 501 is read from a storage at a backup clientfor deduplication storage, first data stream 501 is partitioned intodata regions 504-508 using a partitioning method (e.g., fixed-sizepartitioning) to calculate a checksum for each of the data regions. Thena checksum marker followed by checksum data, represented by checksumblocks 509-513, is appended to each of the data regions 504-508. Theprocess repeats until all data are processed, which forms a second datastream 502 with interleaving checksum blocks 509-513. Second data stream502 is then transmitted from the client to a remote storage fordeduplication. The storage system can then recognize the checksummarkers embedded within checksum blocks 509-513 and pull those markersand checksum data out of data stream 502 to recover the original datastream 501, where the original data stream 501 is then deduplicated,where checksum blocks 509-513 may be separately stored.

A checksum or hash sum is a fixed-size datum computed from an arbitraryblock of digital data for the purpose of detecting accidental errorsthat may have been introduced during its transmission or storage. Theintegrity of the data can be checked at any later time by recomputingthe checksum and comparing it with the stored one. If the checksumsmatch, the data was likely not accidentally altered. The procedure thatyields the checksum from the data is called a checksum function orchecksum algorithm. A good checksum algorithm will yield a differentresult with high probability when the data is accidentally corrupted; ifthe checksums match, the data has the same high probability of beingfree of accidental errors.

FIG. 6A is a flow diagram illustrating a method for processing checksummarkers according to one embodiment of the invention. Method 600 may beperformed by client software 105 of FIG. 1. Referring to FIG. 6, atblock 601, processing logic receives a request for backing up at least aportion of files and/or directories of files of a file system. At block602, processing logic constructs a backup stream having one or more dataregions therein. At block 603, for each of the data region, a checksumis calculated. At block 604, for each of the checksums, a checksummarker is inserted into the backup stream to be associated with each ofthe data regions. At block 605, the backup stream is then transmitted toa backup storage.

FIG. 6B is a flow diagram illustrating a method for processing checksummarkers according to another embodiment of the invention. Method 650 maybe performed by marker processing unit 115 of FIG. 1. Referring to FIG.6B, at block 651, processing logic receives a first data stream from aremote client representing at least a portion of files of the client. Atblock 652, processing logic scans the first data stream to locatechecksum markers that identify the checksums of data regions within thefirst data stream. At block 653, processing logic extracts the checksummarkers and the associated checksums from the first data stream to forma second data stream that is without the checksum markers and thechecksums. At block 604, the second data stream is deduplicated intodeduplicated chunks. At block 605, the deduplicated chunks, as well asthe checksum markers and checksums, are stored separately.

As described above, a backup data stream typically consists of asequence of data objects such as files. Without properly marking dataobject boundaries within the large backup data stream, a storage systemwill not be aware of individual data object boundaries that can be usedas heuristics for segmentation. The segmentation boundaries tend torandomize across file boundaries, which would reduce read performanceand virtual synthetic operation efficiency, because a backup file isoften represented and accessed as a whole unit during backup andsynthetic operations.

According to another embodiment, markers as shown in FIG. 2 can also beused as boundary markers that identify boundaries of data objects withina data stream. The data stream can be deduplicated in view of theboundaries of the data objects for a better deduplication efficiency. Inone embodiment, boundary markers are inserted at natural data objectboundaries into backup data stream at backup clients, which can berecognized by a storage system to improve performance and efficiency.Armed with such knowledge on data object layout boundaries, the storagesystem can produce internal segmentation boundaries that align withnatural file boundaries

FIG. 7 is a block diagram illustrating a process for handling boundarymarkers according to one embodiment of the invention. Referring to FIG.7, when a storage system receives data stream 701 from a remote clientfor deduplication, the storage system performs a chunking operation tosegment data stream 701 into chunks using a predetermined chunkingalgorithm. In this example, data stream 701 includes data objects703-707 encapsulated therein, where a data object may represent a fileor a group of files (e.g., directory of files). During chunkingoperations, a chunking unit of a deduplication storage engine (e.g.,deduplication storage engine 107 of FIG. 1) parses data stream 701 toanchor, using a predetermined anchoring algorithm, multiple anchorpoints such as anchor points 711-718 indicating the boundaries of thechunks to be formed.

In one embodiment, such an anchoring algorithm may be a type-9 anchoringalgorithm. A type-9 anchoring algorithm produces segments in sizesbetween a minimum of 4 kilobytes (KB) and a maximum of 12 KB with anaverage size of around 8 KB. It computes a rolling 32-bit exclusive OR(XOR) checksum up to 12 KB in data stream and selects the byte beyond 4KB as an anchoring point where it produces a maximum checksum value.

In this example, without aware of data object boundaries, the anchoringalgorithm produces anchoring points 711-718. As shown in FIG. 7, thechunk anchored by anchoring points 712 and 713 includes content of bothdata objects 703-704. Similarly, the chunks anchored between anchoringpoints 713-714, between anchoring points 715-716, and between anchoringpoints 716-717 also include content of multiple data objects. Asdescribed above, content of chunks associated with the same data objecttends to be accessed as a whole. If a chunk includes content of multipledata objects, the efficiency of accessing the chunks associated with aparticular data object will be significantly impacted.

With embedded boundary markers 731-735 as shown in data stream 702, theposition of data object boundaries can be determined within data stream702. As a result, when an anchoring point is determined using anexisting anchoring algorithm such as type-9 anchoring algorithm, theboundary markers can be used to calculate a distance between the closestnatural data object boundaries identified through boundary markers fromthe anchoring points. According to one embodiment, if such a distance iswithin a pre-defined threshold value, one of closest natural data objectboundaries that best match other existing segmentation policies ischosen as an actual anchoring point for segmentation. The segmentationprocess proceeds with the rest of backup data stream starting from thenewly established anchoring point. Referring to FIG. 7, with some ofanchor points 721-728, such as anchor points 722, 724, 726, and 727 areselected to be aligned with data object boundaries identified byboundary markers 731-734, respectively. As a result, no chunk containsthe content of multiple data objects or a number of chunks that containcontent of multiple data objects can be reduced.

As segments (e.g., segment 750) are being generated through thesegmentation process, they are being filtered for duplicate removal,then compressed and packed into containers for storage in compressionunit called compression region or CR (e.g., compression region 751).Such containers (e.g., container 752) are appended to a container log,and each compression region contains a number of segments. In additionto alignment of anchoring points with natural data object boundariesduring segmentation process, after segments have been filtered forremoval of duplicates, segments belonging to a same file are to bepacked into a single compression region and a single container forstorage if there is no size constraint and other system policies permit.With such container packing optimization, many relatively small data canbe retrieved with much higher probability that loading data only from asingle container or a single compression region from a single containeris sufficient, thus read performance can be improved.

FIG. 8A is a flow diagram illustrating a method for processing boundarymarkers according to one embodiment of the invention. Method 800 may beperformed by client software 105 of FIG. 1. Referring to FIG. 8A,processing logic receives a request to back up at least a portion offiles and/or directories of a file system. At block 802, a backup streamis constructed including one or more data objects representing the filesand/or directories. At block 803, processing logic identifies a dataobject boundary for each of the data objects within the data stream. Atblock 804, for each of the data object boundaries, a boundary marker isinserted to identify the corresponding data object boundary. At block805, the data stream having the boundary markers therein is thentransmitted to a remote storage system.

FIG. 8B is a flow diagram illustrating a method for processing boundarymarkers according to another embodiment of the invention. Method 850 maybe performed by marker processing unit 115 of FIG. 1. Referring to FIG.8B, at block 851, processing logic receives from a client a data streamhaving data objects representing at least a portion of files and/ordirectories of files of the client. At block 852, processing logic scansthe data stream to recognize and locate boundary markers that identifyboundaries of the data objects within the data stream. At block 853, thedata stream is deduplicated in view of the data object boundaries thatare identified by the boundary markers. Note that unlike checksummarkers, boundary markers may not need to be removed from the datastream prior to deduplication. A boundary marker tends to be a constantwith a small size (e.g., an integer). Such boundary markers do not havea big impact on the deduplication efficiency. Alternatively, theboundary markers can be removed prior to the deduplication.

FIG. 9 is a block diagram illustrating a deduplication storage systemaccording to one embodiment of the invention. For example, deduplicationstorage system 1000 may be implemented as part of a deduplicationstorage system as described above, such as storage system 104 of FIG. 1.In one embodiment, storage system 1000 may represent a file server(e.g., an appliance used to provide network attached storage (NAS)capability), a block-based storage server (e.g., used to provide SANcapability), a unified storage device (e.g., one which combines NAS andSAN capabilities), a nearline storage device, a direct attached storage(DAS) device, a tape backup device, or essentially any other type ofdata storage device. Storage system 1000 may have a distributedarchitecture, or all of its components may be integrated into a singleunit. Storage system 1000 may be implemented as part of an archiveand/or backup system such as a deduplicating storage system availablefrom EMC® Corporation of Hopkinton, Mass.

In one embodiment, storage system 1000 includes a deduplication engine1001 interfacing one or more clients 1014 with one or more storage units1010 storing metadata 1016 and data objects 1018. Clients 1014 may beany kinds of clients such as a client application or backup softwarelocated locally or remotely over a network. A network may be any type ofnetworks such as a local area network (LAN), a wide area network (WAN)such as the Internet, a corporate intranet, a metropolitan area network(MAN), a storage area network (SAN), a bus, or a combination thereof,wired and/or wireless.

Storage units 1010 may be implemented locally (e.g., single nodeoperating environment) or remotely (e.g., multi-node operatingenvironment) via an interconnect, which may be a bus and/or a network.In one embodiment, one of storage units 1010 operates as an activestorage to receive and store external or fresh user data, while theanother one of storage units 1010 operates as a target storage unit toperiodically archive data from the active storage unit according to anarchiving policy or scheme. Storage units 1010 may be, for example,conventional magnetic disks, optical disks such as CD-ROM or DVD basedstorage, magnetic tape storage, magneto-optical (MO) storage media,solid state disks, flash memory based devices, or any other type ofnon-volatile storage devices suitable for storing large volumes of data.Storage units 108-109 may also be combinations of such devices. In thecase of disk storage media, the storage units 1010 may be organized intoone or more volumes of Redundant Array of Inexpensive Disks (RAID). Datastored in the storage units may be stored in a compressed form (e.g.,lossless compression: Huffman coding, Lempel-Ziv Welch coding; deltaencoding: a reference to a chunk plus a difference; etc.). In oneembodiment, different storage units may use different compressionmethods (e.g., main or active storage unit from other storage units, onestorage unit from another storage unit, etc.).

The metadata, such as metadata 1016, may be stored in at least some ofstorage units 1010, such that files can be accessed independent ofanother storage unit. Metadata of each storage unit includes enoughinformation to provide access to the files it contains. In oneembodiment, metadata may include fingerprints contained within dataobjects 1018, where a data object may represent a data chunk, a CR ofdata chunks, or a container of one or more CRs. Fingerprints are mappedto a particular data object via metadata 1016, enabling the system toidentify the location of the data object containing a chunk representedby a particular fingerprint. When an active storage unit fails, metadatacontained in another storage unit may be utilized to recover the activestorage unit. When one storage unit is unavailable (e.g., the storageunit has failed, or is being upgraded, etc.), the system remains up toprovide access to any file not stored in the failed storage unit. When afile is deleted, the metadata associated with the files in the system isupdated to reflect that the file has been deleted.

In one embodiment, the metadata information includes a file name, astorage unit where the chunks associated with the file name are stored,reconstruction information for the file using the chunks, and any otherappropriate metadata information. In one embodiment, a copy of themetadata is stored on a storage unit for files stored on a storage unitso that files that are stored on the storage unit can be accessed usingonly the information stored on the storage unit. In one embodiment, amain set of metadata information can be reconstructed by usinginformation of other storage units associated with the storage system inthe event that the main metadata is lost, corrupted, damaged, etc.Metadata for a storage unit can be reconstructed using metadatainformation stored on a main storage unit or other storage unit (e.g.,replica storage unit). Metadata information further includes indexinformation (e.g., location information for chunks in storage units,identifying specific data objects).

In one embodiment, deduplication storage engine 1001 includes fileservice interface 1002, segmenter 1004, duplicate eliminator 1006, filesystem control 1008, and storage unit interface 1012. Deduplicationstorage engine 1001 receives a file or files (or data item(s)) via fileservice interface 1002, which may be part of a file system namespace ofa file system associated with the deduplication storage engine 1001. Thefile system namespace refers to the way files are identified andorganized in the system. An example is to organize the fileshierarchically into directories or folders. File service interface 1012supports a variety of protocols, including a network file system (NFS),a common Internet file system (CIFS), and a virtual tape libraryinterface (VTL), etc.

The file(s) is/are processed by segmenter 1004 and file system control1008. Segmenter 1004 breaks the file(s) into variable-length chunksbased on a variety of rules or considerations. For example, the file(s)may be broken into chunks by identifying chunk boundaries using acontent-based technique (e.g., a function is calculated at variouslocations of a file, when the function is equal to a value or when thevalue is a minimum, a maximum, or other value relative to other functionvalues calculated for the file), a non-content-based technique (e.g.,based on size of the chunk), or any other appropriate technique. In oneembodiment, a chunk is restricted to a minimum and/or maximum length, toa minimum or maximum number of chunks per file, or any other appropriatelimitation.

In one embodiment, file system control 1008 processes information toindicate the chunk(s) association with a file. In some embodiments, alist of fingerprints is used to indicate chunk(s) associated with afile. File system control 1008 passes chunk association information(e.g., representative data such as a fingerprint) to an index (notshown). The index is used to locate stored chunks in storage units 1010via storage unit interface 1012. Duplicate eliminator 1006 identifieswhether a newly received chunk has already been stored in storage units1010. In the event that a chunk has already been stored in storageunit(s), a reference to the previously stored chunk is stored, forexample, in a chunk tree associated with the file, instead of storingthe newly received chunk. A chunk tree of a file may include one or morenodes and each node represents or references one of the deduplicatedchunks stored in storage units 1010 that make up the file. Chunks arethen packed by a container manager (not shown) into one or more storagecontainers stored in storage units 1010. The deduplicated chunks may befurther compressed using a variation of compression algorithms, such asa Lempel-Ziv algorithm before being stored.

When a file is to be retrieved, file service interface 1002 isconfigured to communicate with file system control 1008 to identifyappropriate chunks stored in storage units 1010 via storage unitinterface 1012. Storage unit interface 1012 may be implemented as partof a container manager. File system control 1008 communicates with anindex (not shown) to locate appropriate chunks stored in storage unitsvia storage unit interface 1012. Appropriate chunks are retrieved fromthe associated containers via the container manager and are used toconstruct the requested file. The file is provided via interface 1002 inresponse to the request. In one embodiment, file system control 1008utilizes a tree (e.g., a chunk tree) of content-based identifiers (e.g.,fingerprints) to associate a file with data chunks and their locationsin storage unit(s). In the event that a chunk associated with a givenfile or file changes, the content-based identifiers will change and thechanges will ripple from the bottom to the top of the tree associatedwith the file efficiently since the appropriate content-basedidentifiers are easily identified using the tree structure. Note thatsome or all of the components as shown as part of deduplication engine1001 may be implemented in software, hardware, or a combination thereof.For example, deduplication engine 1001 may be implemented in a form ofexecutable instructions that can be stored in a machine-readable storagemedium, where the instructions can be executed in a memory by aprocessor.

In one embodiment, storage system 1000 may be used as a tier of storagein a storage hierarchy that comprises other tiers of storage. One ormore tiers of storage in this hierarchy may utilize different kinds ofstorage devices and/or may be optimized for different characteristicssuch as random update performance. Files are periodically moved amongthe tiers based on data management policies to achieve a cost-effectivematch to the current storage requirements of the files. For example, afile may initially be stored in a tier of storage that offers highperformance for reads and writes. As the file ages, it may be moved intoa tier of storage according to one embodiment of the invention. Invarious embodiments, tiers include different storage technologies (e.g.,tape, hard drives, semiconductor-based memories, optical drives, etc.),different locations (e.g., local computer storage, local networkstorage, remote network storage, distributed storage, cloud storage,archive storage, vault storage, etc.), or any other appropriate storagefor a tiered data storage system.

Some portions of the preceding detailed descriptions have been presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. These algorithmic descriptions andrepresentations are the ways used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as those set forth in the claims below, refer to the actionand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Embodiments of the invention also relate to an apparatus for performingthe operations herein. Such a computer program is stored in anon-transitory computer readable medium. A machine-readable mediumincludes any mechanism for storing information in a form readable by amachine (e.g., a computer). For example, a machine-readable (e.g.,computer-readable) medium includes a machine (e.g., a computer) readablestorage medium (e.g., read only memory (“ROM”), random access memory(“RAM”), magnetic disk storage media, optical storage media, flashmemory devices).

The processes or methods depicted in the preceding figures may beperformed by processing logic that comprises hardware (e.g. circuitry,dedicated logic, etc.), software (e.g., embodied on a non-transitorycomputer readable medium), or a combination of both. Although theprocesses or methods are described above in terms of some sequentialoperations, it should be appreciated that some of the operationsdescribed may be performed in a different order. Moreover, someoperations may be performed in parallel rather than sequentially.

Embodiments of the present invention are not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof embodiments of the invention as described herein.

In the foregoing specification, embodiments of the invention have beendescribed with reference to specific exemplary embodiments thereof. Itwill be evident that various modifications may be made thereto withoutdeparting from the broader spirit and scope of the invention as setforth in the following claims. The specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense.

What is claimed is:
 1. A computer-implemented method for deduplicatingdata, comprising: receiving at a storage system over a network from aclient a first data stream having a plurality of data regions and aplurality of checksums for verifying integrity of the data regionsembedded therein, the first data stream representing a file or adirectory of one or more files of a file system associated with theclient; scanning the first data stream to recognize a plurality ofchecksum markers that identify the checksums, wherein the checksummarkers were inserted into the first data stream by the client prior toreceiving the first data stream over the network; extracting thechecksum markers and the checksums from the first data stream togenerate second data stream without the checksum markers and associatedchecksum data therein; and deduplicating the second data stream into aplurality of deduplicated chunks.
 2. The method of claim 1, furthercomprising separately storing the deduplicated chunks and the checksummarkers and the associated checksums in a storage device.
 3. The methodof claim 1, wherein each checksum marker includes a predeterminedpattern and a length field indicating a size of the associated checksumthat immediately follows the associated checksum marker, and whereineach checksum marker is recognized by matching the predeterminedpattern.
 4. The method of claim 3, wherein the predetermined patternincludes a predetermined number.
 5. A computer-implemented method fordeduplicating data, comprising: receiving from a client a first datastream having a plurality of data regions and a plurality of checksumsfor verifying integrity of the data regions embedded therein, the firstdata stream representing a file or a directory of one or more files of afile system associated with the client; scanning the first data streamto recognize a plurality of checksum markers that identify thechecksums; extracting the checksum markers and the checksums from thefirst data stream to generate second data stream without the checksummarkers and associated checksums therein; and deduplicating the seconddata stream into a plurality of deduplicated chunks; and separatelystoring the deduplicated chunks and the checksum markers and theassociated checksums in a storage device, wherein the separately storedchecksum markers and the associated checksums are to be incorporatedwith the deduplicated chunks during a restoration of the first datastream subsequently.
 6. A computer-implemented method for deduplicatingdata, comprising: receiving from a client a first data stream having aplurality of data regions and a plurality of checksums identified by aplurality of checksum markers for verifying integrity of the dataregions embedded therein, the first data stream representing a file or adirectory of one or more files of a file system associated with theclient, wherein each checksum marker includes a predetermined pattern alength field indicating a size of the associated checksum thatimmediately follows the associated checksum marker, wherein eachchecksum marker is recognized by matching the predetermined pattern, andwherein each checksum marker further includes a type field specifyingthat the corresponding marker is a checksum marker that is differentfrom other types of markers; and deduplicating the first data streamwith the checksums removed into a plurality of deduplicated chunks.
 7. Anon-transitory machine-readable medium having instructions storedtherein, which when executed by a processor, cause the processor toperform operations for deduplicating data, the operations comprising:receiving at a storage system over a network from a client a first datastream having a plurality of data regions and a plurality of checksumsfor verifying integrity of the data regions embedded therein, the firstdata stream representing a file or a directory of one or more files of afile system associated with the client; scanning the first data streamto recognize a plurality of checksum markers that identify thechecksums, wherein the checksum markers were inserted into the firstdata stream by the client prior to receiving the first data stream overthe network; extracting the checksum markers and the checksums from thefirst data stream to generate a second data stream without the checksummarkers and associated checksum data therein; and deduplicating thesecond data stream into a plurality of deduplicated chunks.
 8. Thenon-transitory machine-readable medium of claim 7, wherein theoperations further comprise separately storing the deduplicated chunksand the checksum markers and the associated checksums in a storagedevice.
 9. The non-transitory machine-readable medium of claim 7,wherein each checksum marker includes a predetermined pattern and alength field indicating a size of the associated checksum thatimmediately follows the associated checksum marker, and wherein eachchecksum marker is recognized by matching the predetermined pattern. 10.The non-transitory machine-readable medium of claim 9, wherein thepredetermined pattern includes a predetermined number.
 11. Anon-transitory machine-readable medium having instructions storedtherein, which when executed by a processor, cause the processor toperform operations for deduplicating data, the operations comprising:receiving from a client a first data stream having a plurality of dataregions and a plurality of checksums for verifying integrity of the dataregions embedded therein, the first data stream representing a file or adirectory of one or more files of a file system associated with theclient; scanning the first data stream to recognize a plurality ofchecksum markers that identify the checksums; extracting the checksummarkers and the checksums from the first data stream to generate asecond data stream without the checksum markers and associated checksumstherein; and deduplicating the second data stream into a plurality ofdeduplicated chunks; and separately storing the deduplicated chunks andthe checksum markers and the associated checksums in a storage device,wherein the separately stored checksum markers and the associatedchecksums are to be incorporated with the deduplicated chunks during arestoration of the first data stream subsequently.
 12. A non-transitorymachine-readable medium having instructions stored therein, which whenexecuted by a processor, cause the processor to perform operations fordeduplicating data, the operations comprising: receiving from a client afirst data stream having a plurality of data regions and a plurality ofchecksums identified by a plurality of checksum markers for verifyingintegrity of the data regions embedded therein, the first data streamrepresenting a file or a directory of one or more files of a file systemassociated with the client, wherein each checksum marker includes apredetermined pattern a length field indicating a size of the associatedchecksum that immediately follows the associated checksum marker,wherein each checksum marker is recognized by matching the predeterminedpattern, and wherein each checksum marker further includes a type fieldspecifying that the corresponding marker is a checksum marker that isdifferent from other types of markers; and deduplicating the first datastream with the checksums removed into a plurality of deduplicatedchunks.
 13. A data processing system, comprising: a processor; and amemory coupled to the processor to store instructions, which whenexecuted from the memory, cause the processor to receive from a clientover a network a first data stream having a plurality of data regionsand a plurality of checksums for verifying integrity of the data regionsembedded therein, the first data stream representing a file or adirectory of one or more files of a file system associated with theclient, scan the first data stream to recognize a plurality of checksummarkers that identify the checksums, wherein the checksum markers wereinserted into the first data stream by the client prior to receiving thefirst data stream over the network, extract the checksum markers and thechecksums from the first data stream to generate a second data streamwithout the checksum markers and associated checksum data therein, anddeduplicate the second data stream into a plurality of deduplicatedchunks.
 14. The system of claim 13, wherein the deduplicated chunks andthe checksum markers and the associated checksums are separately storedin a storage device.
 15. The system of claim 13, wherein each checksummarker includes a predetermined pattern and a length field indicating asize of the associated checksum that immediately follows the associatedchecksum marker, and wherein each checksum marker is recognized bymatching the predetermined pattern.
 16. The system of claim 15, whereinthe predetermined pattern includes a predetermined number.
 17. A dataprocessing system, comprising: a processor; and a memory coupled to theprocessor to store instructions, which when executed from the memory,cause the processor to receive from a client a first data stream havinga plurality of data regions and a plurality of checksums for verifyingintegrity of the data regions embedded therein, the first data streamrepresenting a file or a directory of one or more files of a file systemassociated with the client, scan the first data stream to recognize aplurality of checksum markers that identify the checksums, extract thechecksum markers and the checksums from the first data stream generate asecond data stream without the checksum markers and associated checksumstherein, deduplicate the second data stream into a plurality ofdeduplicated chunks, and separately store the deduplicated chunks andthe checksum markers and the associated checksums in a storage device,wherein the separately stored checksum markers and the associatedchecksums are to be incorporated with the deduplicated chunks during arestoration of the first data stream subsequently.
 18. A data processingsystem, comprising: a processor; and a memory coupled to the processorto store instructions, which when executed from the memory, cause theprocessor to receive from a client a first data stream having aplurality of data regions and a plurality of checksums identified by aplurality of checksum markers for verifying integrity of the dataregions embedded therein, the first data stream representing a file or adirectory of one or more files of a file system associated with theclient, each checksum marker includes a predetermined pattern and alength field indicating a size of the associated checksum thatimmediately follows the associated checksum marker, wherein eachchecksum marker is recognized by matching the predetermined pattern, andwherein each checksum marker further includes a type field specifyingthat the corresponding marker is a checksum marker that is differentfrom other types of markers, and deduplicate the first data stream withthe checksums removed into a plurality of deduplicated chunks.